Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI: Extending follow-up #2587

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

AI: Extending follow-up #2587

wants to merge 2 commits into from

Conversation

mnocon
Copy link
Contributor

@mnocon mnocon commented Jan 2, 2025

Target: 4.6, master

Things done:

  1. Applied review remarks from Extending AI Actions #2537 (except the Action Configuration example - I've kept it as is to introduce Refine text there)

Previews:

@mnocon mnocon changed the title Ai follow up AI: Extending follow-up and new REST endpoints Feb 6, 2025
@mnocon mnocon marked this pull request as ready for review February 6, 2025 11:40
@mnocon mnocon changed the title AI: Extending follow-up and new REST endpoints AI: Extending follow-up Feb 6, 2025
@mnocon mnocon assigned adriendupuis and unassigned adriendupuis Feb 6, 2025
@mnocon mnocon requested a review from adriendupuis February 6, 2025 12:14
}

$id = $this->binaryDataHandler->getIdFromUri($uri);
$file = $this->binaryDataHandler->getContents($id);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably some test here to see if a file is loaded or if URI wasn't corresponding to a file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a specific solution in mind?

I can't use PHP's is_file here (I don't have access to the filepath). The implementation itself will throw BinaryFileNotFoundException if something goes wrong (https://github.com/ibexa/core/blob/main/src/lib/IO/IOBinarydataHandler/Flysystem.php#L66) - catching and rethrowing it seems like an overkill to me

Copy link
Contributor

@adriendupuis adriendupuis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks globaly good. I would change when/how the image URI correctness is checked.

Copy link

Preview of modified Markdown:

Copy link

code_samples/ change report

Before (on target branch)After (in current PR)

code_samples/ai_actions/config/services.yaml

docs/ai_actions/extend_ai_actions.md@50:``` yaml
docs/ai_actions/extend_ai_actions.md@51:[[= include_file('code_samples/ai_actions/config/services.yaml', 25, 28) =]]
docs/ai_actions/extend_ai_actions.md@52:```

001⫶ App\Command\AddMissingAltTextCommand:
002⫶ arguments:

code_samples/ai_actions/config/services.yaml

docs/ai_actions/extend_ai_actions.md@50:``` yaml
docs/ai_actions/extend_ai_actions.md@51:[[= include_file('code_samples/ai_actions/config/services.yaml', 25, 28) =]]
docs/ai_actions/extend_ai_actions.md@52:```

001⫶ App\Command\AddMissingAltTextCommand:
002⫶ arguments:
003⫶            $projectDir: '%kernel.project_dir%'
003⫶            $binaryDataHandler: '@Ibexa\Core\IO\IOBinarydataHandler\SiteAccessDependentBinaryDataHandler'

docs/ai_actions/extend_ai_actions.md@121:``` yaml
docs/ai_actions/extend_ai_actions.md@122:[[= include_file('code_samples/ai_actions/config/services.yaml', 28, 33) =]]
docs/ai_actions/extend_ai_actions.md@123:```

001⫶ App\AI\Handler\LLaVATextToTextActionHandler:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.handler, priority: 0 }
004⫶ - { name: ibexa.ai.action.handler.text_to_text, priority: 0 }

docs/ai_actions/extend_ai_actions.md@141:``` yaml
docs/ai_actions/extend_ai_actions.md@142:[[= include_file('code_samples/ai_actions/config/services.yaml', 34, 41) =]]
docs/ai_actions/extend_ai_actions.md@143:```

001⫶ app.connector_ai.action_configuration.handler.llava_text_to_text.form_mapper.options:
002⫶ class: Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionHandlerOptionsFormMapper
003⫶ arguments:
004⫶ $formType: 'App\Form\Type\TextToTextOptionsType'
005⫶ tags:
006⫶ - name: ibexa.connector_ai.action_configuration.form_mapper.options
007⫶ type: !php/const \App\AI\Handler\LLaVaTextToTextActionHandler::IDENTIFIER


docs/ai_actions/extend_ai_actions.md@121:``` yaml
docs/ai_actions/extend_ai_actions.md@122:[[= include_file('code_samples/ai_actions/config/services.yaml', 28, 33) =]]
docs/ai_actions/extend_ai_actions.md@123:```

001⫶ App\AI\Handler\LLaVATextToTextActionHandler:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.handler, priority: 0 }
004⫶ - { name: ibexa.ai.action.handler.text_to_text, priority: 0 }

docs/ai_actions/extend_ai_actions.md@141:``` yaml
docs/ai_actions/extend_ai_actions.md@142:[[= include_file('code_samples/ai_actions/config/services.yaml', 34, 41) =]]
docs/ai_actions/extend_ai_actions.md@143:```

001⫶ app.connector_ai.action_configuration.handler.llava_text_to_text.form_mapper.options:
002⫶ class: Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionHandlerOptionsFormMapper
003⫶ arguments:
004⫶ $formType: 'App\Form\Type\TextToTextOptionsType'
005⫶ tags:
006⫶ - name: ibexa.connector_ai.action_configuration.form_mapper.options
007⫶ type: !php/const \App\AI\Handler\LLaVaTextToTextActionHandler::IDENTIFIER

docs/ai_actions/extend_ai_actions.md@154:``` yaml
docs/ai_actions/extend_ai_actions.md@155:[[= include_file('code_samples/ai_actions/config/services.yaml', 64, 66) =]]
docs/ai_actions/extend_ai_actions.md@156:```
docs/ai_actions/extend_ai_actions.md@158:``` yaml
docs/ai_actions/extend_ai_actions.md@159:[[= include_file('code_samples/ai_actions/config/services.yaml', 64, 66) =]]
docs/ai_actions/extend_ai_actions.md@160:```

001⫶ Ibexa\Contracts\ConnectorAi\ActionConfiguration\OptionsFormatterInterface:
002⫶ alias: Ibexa\ConnectorAi\ActionConfiguration\JsonOptionsFormatter


001⫶ Ibexa\Contracts\ConnectorAi\ActionConfiguration\OptionsFormatterInterface:
002⫶ alias: Ibexa\ConnectorAi\ActionConfiguration\JsonOptionsFormatter

docs/ai_actions/extend_ai_actions.md@180:``` yaml
docs/ai_actions/extend_ai_actions.md@181:[[= include_file('code_samples/ai_actions/config/services.yaml', 42, 50) =]]
docs/ai_actions/extend_ai_actions.md@182:```
docs/ai_actions/extend_ai_actions.md@184:``` yaml
docs/ai_actions/extend_ai_actions.md@185:[[= include_file('code_samples/ai_actions/config/services.yaml', 42, 50) =]]
docs/ai_actions/extend_ai_actions.md@186:```

001⫶ App\AI\ActionType\TranscribeAudioActionType:
002⫶ arguments:
003⫶ $actionHandlers: !tagged_iterator
004⫶ tag: app.connector_ai.action.handler.audio_to_text
005⫶ default_index_method: getIdentifier
006⫶ index_by: key
007⫶ tags:
008⫶ - { name: ibexa.ai.action.type, identifier: !php/const \App\AI\ActionType\TranscribeAudioActionType::IDENTIFIER }


001⫶ App\AI\ActionType\TranscribeAudioActionType:
002⫶ arguments:
003⫶ $actionHandlers: !tagged_iterator
004⫶ tag: app.connector_ai.action.handler.audio_to_text
005⫶ default_index_method: getIdentifier
006⫶ index_by: key
007⫶ tags:
008⫶ - { name: ibexa.ai.action.type, identifier: !php/const \App\AI\ActionType\TranscribeAudioActionType::IDENTIFIER }

docs/ai_actions/extend_ai_actions.md@218:``` yaml
docs/ai_actions/extend_ai_actions.md@219:[[= include_file('code_samples/ai_actions/config/services.yaml', 51, 58) =]]
docs/ai_actions/extend_ai_actions.md@220:```
docs/ai_actions/extend_ai_actions.md@222:``` yaml
docs/ai_actions/extend_ai_actions.md@223:[[= include_file('code_samples/ai_actions/config/services.yaml', 51, 58) =]]
docs/ai_actions/extend_ai_actions.md@224:```

001⫶ app.connector_ai.action_configuration.handler.transcribe_audio.form_mapper.options:
002⫶ class: Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionTypeOptionsFormMapper
003⫶ arguments:
004⫶ $formType: 'App\Form\Type\TranscribeAudioOptionsType'
005⫶ tags:
006⫶ - name: ibexa.connector_ai.action_configuration.form_mapper.action_type_options
007⫶ type: !php/const \App\AI\ActionType\TranscribeAudioActionType::IDENTIFIER


001⫶ app.connector_ai.action_configuration.handler.transcribe_audio.form_mapper.options:
002⫶ class: Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionTypeOptionsFormMapper
003⫶ arguments:
004⫶ $formType: 'App\Form\Type\TranscribeAudioOptionsType'
005⫶ tags:
006⫶ - name: ibexa.connector_ai.action_configuration.form_mapper.action_type_options
007⫶ type: !php/const \App\AI\ActionType\TranscribeAudioActionType::IDENTIFIER

docs/ai_actions/extend_ai_actions.md@234:``` yaml
docs/ai_actions/extend_ai_actions.md@235:[[= include_file('code_samples/ai_actions/config/services.yaml', 59, 63) =]]
docs/ai_actions/extend_ai_actions.md@236:```
docs/ai_actions/extend_ai_actions.md@238:``` yaml
docs/ai_actions/extend_ai_actions.md@239:[[= include_file('code_samples/ai_actions/config/services.yaml', 59, 63) =]]
docs/ai_actions/extend_ai_actions.md@240:```

001⫶ App\AI\Handler\WhisperAudioToTextActionHandler:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.handler, priority: 0 }
004⫶ - { name: app.connector_ai.action.handler.audio_to_text, priority: 0 }


001⫶ App\AI\Handler\WhisperAudioToTextActionHandler:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.handler, priority: 0 }
004⫶ - { name: app.connector_ai.action.handler.audio_to_text, priority: 0 }

docs/ai_actions/extend_ai_actions.md@252:``` yaml
docs/ai_actions/extend_ai_actions.md@253:[[= include_file('code_samples/ai_actions/config/services.yaml', 68, 72) =]]
docs/ai_actions/extend_ai_actions.md@254:```
docs/ai_actions/extend_ai_actions.md@256:``` yaml
docs/ai_actions/extend_ai_actions.md@257:[[= include_file('code_samples/ai_actions/config/services.yaml', 68, 72) =]]
docs/ai_actions/extend_ai_actions.md@258:```

001⫶ App\AI\REST\Input\Parser\TranscribeAudio:
002⫶ parent: Ibexa\Rest\Server\Common\Parser
003⫶ tags:
004⫶ - { name: ibexa.rest.input.parser, mediaType: application/vnd.ibexa.api.ai.TranscribeAudio }


001⫶ App\AI\REST\Input\Parser\TranscribeAudio:
002⫶ parent: Ibexa\Rest\Server\Common\Parser
003⫶ tags:
004⫶ - { name: ibexa.rest.input.parser, mediaType: application/vnd.ibexa.api.ai.TranscribeAudio }

docs/ai_actions/extend_ai_actions.md@279:``` yaml
docs/ai_actions/extend_ai_actions.md@280:[[= include_file('code_samples/ai_actions/config/services.yaml', 73, 76) =]]
docs/ai_actions/extend_ai_actions.md@281:```
docs/ai_actions/extend_ai_actions.md@283:``` yaml
docs/ai_actions/extend_ai_actions.md@284:[[= include_file('code_samples/ai_actions/config/services.yaml', 73, 76) =]]
docs/ai_actions/extend_ai_actions.md@285:```

001⫶ App\AI\REST\Output\Resolver\AudioTextResolver:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.mime_type, key: application/vnd.ibexa.api.ai.AudioText }


001⫶ App\AI\REST\Output\Resolver\AudioTextResolver:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.mime_type, key: application/vnd.ibexa.api.ai.AudioText }

docs/ai_actions/extend_ai_actions.md@290:``` yaml
docs/ai_actions/extend_ai_actions.md@291:[[= include_file('code_samples/ai_actions/config/services.yaml', 77, 81) =]]
docs/ai_actions/extend_ai_actions.md@292:```
docs/ai_actions/extend_ai_actions.md@294:``` yaml
docs/ai_actions/extend_ai_actions.md@295:[[= include_file('code_samples/ai_actions/config/services.yaml', 77, 81) =]]
docs/ai_actions/extend_ai_actions.md@296:```

001⫶ App\AI\REST\Output\ValueObjectVisitor\AudioText:
002⫶ parent: Ibexa\Contracts\Rest\Output\ValueObjectVisitor
003⫶ tags:
004⫶ - { name: ibexa.rest.output.value_object.visitor, type: App\AI\REST\Value\AudioText }


code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php

docs/ai_actions/extend_ai_actions.md@79:``` php hl_lines="3 17"
docs/ai_actions/extend_ai_actions.md@80:[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 59, 76) =]]
docs/ai_actions/extend_ai_actions.md@81:```

001⫶ $refineTextActionType = $this->actionTypeRegistry->getActionType('refine_text');
002⫶
003⫸ $actionConfigurationCreateStruct = new ActionConfigurationCreateStruct('rewrite_casual');
004⫶
005⫶ $actionConfigurationCreateStruct->setType($refineTextActionType);
006⫶ $actionConfigurationCreateStruct->setName('eng-GB', 'Rewrite in casual tone');
007⫶ $actionConfigurationCreateStruct->setDescription('eng-GB', 'Rewrites the text using a casual tone');
008⫶ $actionConfigurationCreateStruct->setActionHandler('openai-text-to-text');
009⫶ $actionConfigurationCreateStruct->setActionHandlerOptions(new ArrayMap([
010⫶ 'max_tokens' => 4000,
011⫶ 'temperature' => 1,
012⫶ 'prompt' => 'Rewrite this content to improve readability. Preserve meaning and crucial information but use casual language accessible to a broader audience.',
013⫶ 'model' => 'gpt-4-turbo',
014⫶ ]));
015⫶ $actionConfigurationCreateStruct->setEnabled(true);
016⫶
017⫸ $this->actionConfigurationService->createActionConfiguration($actionConfigurationCreateStruct);

docs/ai_actions/extend_ai_actions.md@90:``` php hl_lines="7-8"
docs/ai_actions/extend_ai_actions.md@91:[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]]
docs/ai_actions/extend_ai_actions.md@92:```

001⫶ $action = new RefineTextAction(new Text([

001⫶ App\AI\REST\Output\ValueObjectVisitor\AudioText:
002⫶ parent: Ibexa\Contracts\Rest\Output\ValueObjectVisitor
003⫶ tags:
004⫶ - { name: ibexa.rest.output.value_object.visitor, type: App\AI\REST\Value\AudioText }


code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php

docs/ai_actions/extend_ai_actions.md@79:``` php hl_lines="3 17"
docs/ai_actions/extend_ai_actions.md@80:[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 59, 76) =]]
docs/ai_actions/extend_ai_actions.md@81:```

001⫶ $refineTextActionType = $this->actionTypeRegistry->getActionType('refine_text');
002⫶
003⫸ $actionConfigurationCreateStruct = new ActionConfigurationCreateStruct('rewrite_casual');
004⫶
005⫶ $actionConfigurationCreateStruct->setType($refineTextActionType);
006⫶ $actionConfigurationCreateStruct->setName('eng-GB', 'Rewrite in casual tone');
007⫶ $actionConfigurationCreateStruct->setDescription('eng-GB', 'Rewrites the text using a casual tone');
008⫶ $actionConfigurationCreateStruct->setActionHandler('openai-text-to-text');
009⫶ $actionConfigurationCreateStruct->setActionHandlerOptions(new ArrayMap([
010⫶ 'max_tokens' => 4000,
011⫶ 'temperature' => 1,
012⫶ 'prompt' => 'Rewrite this content to improve readability. Preserve meaning and crucial information but use casual language accessible to a broader audience.',
013⫶ 'model' => 'gpt-4-turbo',
014⫶ ]));
015⫶ $actionConfigurationCreateStruct->setEnabled(true);
016⫶
017⫸ $this->actionConfigurationService->createActionConfiguration($actionConfigurationCreateStruct);

docs/ai_actions/extend_ai_actions.md@90:``` php hl_lines="7-8"
docs/ai_actions/extend_ai_actions.md@91:[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]]
docs/ai_actions/extend_ai_actions.md@92:```

001⫶ $action = new RefineTextAction(new Text([
002⫶            <<<TEXT
003⫶ Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
004⫶ and which usually results in protein folding into a specific 3D structure that determines its activity.
002⫶<<<TEXT
003⫶Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
004⫶and which usually results in protein folding into a specific 3D structure that determines its activity.
005⫶TEXT
006⫶ ]));
007⫸ $actionConfiguration = $this->actionConfigurationService->getActionConfiguration('rewrite_casual');
008⫸ $actionResponse = $this->actionService->execute($action, $actionConfiguration)->getOutput();


code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php

docs/ai_actions/extend_ai_actions.md@16:``` php
005⫶TEXT
006⫶ ]));
007⫸ $actionConfiguration = $this->actionConfigurationService->getActionConfiguration('rewrite_casual');
008⫸ $actionResponse = $this->actionService->execute($action, $actionConfiguration)->getOutput();


code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php

docs/ai_actions/extend_ai_actions.md@16:``` php
docs/ai_actions/extend_ai_actions.md@17:[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php', 101, 120) =]]
docs/ai_actions/extend_ai_actions.md@17:[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php', 102, 121) =]]
docs/ai_actions/extend_ai_actions.md@18:```

001⫶ $action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
002⫶
003⫶ $action->setRuntimeContext(new RuntimeContext(['languageCode' => $languageCode]));
004⫶ $action->setActionContext(
005⫶ new ActionContext(
006⫶ new ActionConfigurationOptions(['default_locale_fallback' => 'en']), // System context
007⫶ new ActionConfigurationOptions(['max_lenght' => 100]), // Action Type options
008⫶ new ActionConfigurationOptions( // Action Handler options
009⫶ [
010⫶ 'prompt' => 'Generate the alt text for this image in less than 100 characters.',
011⫶ 'temperature' => 0.7,
012⫶ 'max_tokens' => 4096,
013⫶ 'model' => 'gpt-4o-mini',
014⫶ ]
015⫶ )
016⫶ )
017⫶ );
018⫶
019⫶ $output = $this->actionService->execute($action)->getOutput();

docs/ai_actions/extend_ai_actions.md@18:```

001⫶ $action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
002⫶
003⫶ $action->setRuntimeContext(new RuntimeContext(['languageCode' => $languageCode]));
004⫶ $action->setActionContext(
005⫶ new ActionContext(
006⫶ new ActionConfigurationOptions(['default_locale_fallback' => 'en']), // System context
007⫶ new ActionConfigurationOptions(['max_lenght' => 100]), // Action Type options
008⫶ new ActionConfigurationOptions( // Action Handler options
009⫶ [
010⫶ 'prompt' => 'Generate the alt text for this image in less than 100 characters.',
011⫶ 'temperature' => 0.7,
012⫶ 'max_tokens' => 4096,
013⫶ 'model' => 'gpt-4o-mini',
014⫶ ]
015⫶ )
016⫶ )
017⫶ );
018⫶
019⫶ $output = $this->actionService->execute($action)->getOutput();

docs/ai_actions/extend_ai_actions.md@46:``` php hl_lines="87 100-125"
docs/ai_actions/extend_ai_actions.md@46:``` php hl_lines="88 101-126"
docs/ai_actions/extend_ai_actions.md@47:[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php') =]]
docs/ai_actions/extend_ai_actions.md@48:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\Command;
006⫶
007⫶use Ibexa\Contracts\ConnectorAi\Action\ActionContext;
008⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Image;
009⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Text;
010⫶use Ibexa\Contracts\ConnectorAi\Action\GenerateAltTextAction;
011⫶use Ibexa\Contracts\ConnectorAi\Action\RuntimeContext;
012⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\ActionConfigurationOptions;
013⫶use Ibexa\Contracts\ConnectorAi\ActionServiceInterface;
014⫶use Ibexa\Contracts\Core\Repository\ContentService;
015⫶use Ibexa\Contracts\Core\Repository\FieldTypeService;
016⫶use Ibexa\Contracts\Core\Repository\PermissionResolver;
017⫶use Ibexa\Contracts\Core\Repository\UserService;
018⫶use Ibexa\Contracts\Core\Repository\Values\Content\ContentList;
019⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\ContentTypeIdentifier;
020⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\DateMetadata;
021⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\Operator;
022⫶use Ibexa\Contracts\Core\Repository\Values\Filter\Filter;
023⫶use Ibexa\Core\FieldType\Image\Value;
docs/ai_actions/extend_ai_actions.md@47:[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php') =]]
docs/ai_actions/extend_ai_actions.md@48:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\Command;
006⫶
007⫶use Ibexa\Contracts\ConnectorAi\Action\ActionContext;
008⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Image;
009⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Text;
010⫶use Ibexa\Contracts\ConnectorAi\Action\GenerateAltTextAction;
011⫶use Ibexa\Contracts\ConnectorAi\Action\RuntimeContext;
012⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\ActionConfigurationOptions;
013⫶use Ibexa\Contracts\ConnectorAi\ActionServiceInterface;
014⫶use Ibexa\Contracts\Core\Repository\ContentService;
015⫶use Ibexa\Contracts\Core\Repository\FieldTypeService;
016⫶use Ibexa\Contracts\Core\Repository\PermissionResolver;
017⫶use Ibexa\Contracts\Core\Repository\UserService;
018⫶use Ibexa\Contracts\Core\Repository\Values\Content\ContentList;
019⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\ContentTypeIdentifier;
020⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\DateMetadata;
021⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\Operator;
022⫶use Ibexa\Contracts\Core\Repository\Values\Filter\Filter;
023⫶use Ibexa\Core\FieldType\Image\Value;
024⫶use Symfony\Component\Console\Command\Command;
025⫶use Symfony\Component\Console\Input\InputArgument;
026⫶use Symfony\Component\Console\Input\InputInterface;
027⫶use Symfony\Component\Console\Output\OutputInterface;
028⫶
029⫶final class AddMissingAltTextCommand extends Command
030⫶{
031⫶ protected static $defaultName = 'app:add-alt-text';
032⫶
033⫶ private const IMAGE_FIELD_IDENTIFIER = 'image';
034⫶
035⫶ private ContentService $contentService;
036⫶
037⫶ private PermissionResolver $permissionResolver;
038⫶
039⫶ private UserService $userService;
040⫶
041⫶ private FieldTypeService $fieldTypeService;
042⫶
043⫶ private ActionServiceInterface $actionService;
044⫶
045⫶ private string $projectDir;
046⫶
047⫶ public function __construct(
048⫶ ContentService $contentService,
049⫶ PermissionResolver $permissionResolver,
050⫶ UserService $userService,
051⫶ FieldTypeService $fieldTypeService,
052⫶ ActionServiceInterface $actionService,
053⫶ string $projectDir
054⫶ ) {
055⫶ parent::__construct();
056⫶ $this->contentService = $contentService;
057⫶ $this->permissionResolver = $permissionResolver;
058⫶ $this->userService = $userService;
059⫶ $this->fieldTypeService = $fieldTypeService;
060⫶ $this->actionService = $actionService;
061⫶ $this->projectDir = $projectDir;
062⫶ }
063⫶
064⫶ protected function configure(): void
065⫶ {
066⫶ $this->addArgument('user', InputArgument::OPTIONAL, 'Login of the user executing the actions', 'admin');
067⫶ }
068⫶
069⫶ protected function execute(InputInterface $input, OutputInterface $output): int
070⫶ {
071⫶ $this->setUser($input->getArgument('user'));
072⫶
073⫶ $modifiedImages = $this->getModifiedImages();
074⫶ $output->writeln(sprintf('Found %d modified image in the last 24h', $modifiedImages->getTotalCount()));
075⫶
076⫶ /** @var \Ibexa\Contracts\Core\Repository\Values\Content\Content $content */
077⫶ foreach ($modifiedImages as $content) {
078⫶ /** @var ?Value $value */
079⫶ $value = $content->getFieldValue(self::IMAGE_FIELD_IDENTIFIER);
080⫶
081⫶ if ($value === null || !$this->shouldGenerateAltText($value)) {
082⫶ $output->writeln(sprintf('Image %s has the image field empty or the alternative text is already specified. Skipping.', $content->getName()));
083⫶ continue;
084⫶ }
085⫶
086⫶ $contentUpdateStruct = $this->contentService->newContentUpdateStruct();
087⫸ $value->alternativeText = $this->getSuggestedAltText($this->convertImageToBase64($value->uri), $content->getDefaultLanguageCode());
088⫶ $contentUpdateStruct->setField(self::IMAGE_FIELD_IDENTIFIER, $value);
089⫶
090⫶ $updatedContent = $this->contentService->updateContent(
091⫶ $this->contentService->createContentDraft($content->getContentInfo())->getVersionInfo(),
092⫶ $contentUpdateStruct
093⫶ );
094⫶ $this->contentService->publishVersion($updatedContent->getVersionInfo());
095⫶ }
096⫶
097⫶ return Command::SUCCESS;
098⫶ }
099⫶
100⫸ private function getSuggestedAltText(string $imageEncodedInBase64, string $languageCode): string
101⫸ {
102⫸ $action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
103⫸
104⫸ $action->setRuntimeContext(new RuntimeContext(['languageCode' => $languageCode]));
105⫸ $action->setActionContext(
106⫸ new ActionContext(
107⫸ new ActionConfigurationOptions(['default_locale_fallback' => 'en']), // System context
108⫸ new ActionConfigurationOptions(['max_lenght' => 100]), // Action Type options
109⫸ new ActionConfigurationOptions( // Action Handler options
110⫸ [
111⫸ 'prompt' => 'Generate the alt text for this image in less than 100 characters.',
112⫸ 'temperature' => 0.7,
113⫸ 'max_tokens' => 4096,
114⫸ 'model' => 'gpt-4o-mini',
115⫸ ]
116⫸ )
117⫸ )
118⫸ );
119⫸
120⫸ $output = $this->actionService->execute($action)->getOutput();
121⫸
122⫸ assert($output instanceof Text);
123⫸
124⫸ return $output->getText();
125⫸ }
126⫶
127⫶ private function convertImageToBase64(?string $uri): string
128⫶ {
129⫶ $file = file_get_contents($this->projectDir . \DIRECTORY_SEPARATOR . 'public' . \DIRECTORY_SEPARATOR . $uri);
130⫶ if ($file === false) {
131⫶ throw new \RuntimeException('Cannot read file');
132⫶ }
133⫶
134⫶ return 'data:image/jpeg;base64,' . base64_encode($file);
135⫶ }
136⫶
137⫶ private function getModifiedImages(): ContentList
138⫶ {
139⫶ $filter = (new Filter())
140⫶ ->withCriterion(
141⫶ new DateMetadata(DateMetadata::MODIFIED, Operator::GTE, strtotime('-1 day'))
142⫶ )
143⫶ ->andWithCriterion(new ContentTypeIdentifier('image'));
144⫶
145⫶ return $this->contentService->find($filter);
146⫶ }
147⫶
024⫶use Ibexa\Core\IO\IOBinarydataHandler;
025⫶use Symfony\Component\Console\Command\Command;
026⫶use Symfony\Component\Console\Input\InputArgument;
027⫶use Symfony\Component\Console\Input\InputInterface;
028⫶use Symfony\Component\Console\Output\OutputInterface;
029⫶
030⫶final class AddMissingAltTextCommand extends Command
031⫶{
032⫶ protected static $defaultName = 'app:add-alt-text';
033⫶
034⫶ private const IMAGE_FIELD_IDENTIFIER = 'image';
035⫶
036⫶ private ContentService $contentService;
037⫶
038⫶ private PermissionResolver $permissionResolver;
039⫶
040⫶ private UserService $userService;
041⫶
042⫶ private FieldTypeService $fieldTypeService;
043⫶
044⫶ private ActionServiceInterface $actionService;
045⫶
046⫶ private IOBinarydataHandler $binaryDataHandler;
047⫶
048⫶ public function __construct(
049⫶ ContentService $contentService,
050⫶ PermissionResolver $permissionResolver,
051⫶ UserService $userService,
052⫶ FieldTypeService $fieldTypeService,
053⫶ ActionServiceInterface $actionService,
054⫶ IOBinarydataHandler $binaryDataHandler
055⫶ ) {
056⫶ parent::__construct();
057⫶ $this->contentService = $contentService;
058⫶ $this->permissionResolver = $permissionResolver;
059⫶ $this->userService = $userService;
060⫶ $this->fieldTypeService = $fieldTypeService;
061⫶ $this->actionService = $actionService;
062⫶ $this->binaryDataHandler = $binaryDataHandler;
063⫶ }
064⫶
065⫶ protected function configure(): void
066⫶ {
067⫶ $this->addArgument('user', InputArgument::OPTIONAL, 'Login of the user executing the actions', 'admin');
068⫶ }
069⫶
070⫶ protected function execute(InputInterface $input, OutputInterface $output): int
071⫶ {
072⫶ $this->setUser($input->getArgument('user'));
073⫶
074⫶ $modifiedImages = $this->getModifiedImages();
075⫶ $output->writeln(sprintf('Found %d modified image in the last 24h', $modifiedImages->getTotalCount()));
076⫶
077⫶ /** @var \Ibexa\Contracts\Core\Repository\Values\Content\Content $content */
078⫶ foreach ($modifiedImages as $content) {
079⫶ /** @var \Ibexa\Core\FieldType\Image\Value $value */
080⫶ $value = $content->getFieldValue(self::IMAGE_FIELD_IDENTIFIER);
081⫶
082⫶ if ($value === null || !$this->shouldGenerateAltText($value)) {
083⫶ $output->writeln(sprintf('Image %s has the image field empty, the file cannot be accessed, or the alternative text is already specified. Skipping.', $content->getName()));
084⫶ continue;
085⫶ }
086⫶
087⫶ $contentUpdateStruct = $this->contentService->newContentUpdateStruct();
088⫸ $value->alternativeText = $this->getSuggestedAltText($this->convertImageToBase64($value->uri), $content->getDefaultLanguageCode());
089⫶ $contentUpdateStruct->setField(self::IMAGE_FIELD_IDENTIFIER, $value);
090⫶
091⫶ $updatedContent = $this->contentService->updateContent(
092⫶ $this->contentService->createContentDraft($content->getContentInfo())->getVersionInfo(),
093⫶ $contentUpdateStruct
094⫶ );
095⫶ $this->contentService->publishVersion($updatedContent->getVersionInfo());
096⫶ }
097⫶
098⫶ return Command::SUCCESS;
099⫶ }
100⫶
101⫸ private function getSuggestedAltText(string $imageEncodedInBase64, string $languageCode): string
102⫸ {
103⫸ $action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
104⫸
105⫸ $action->setRuntimeContext(new RuntimeContext(['languageCode' => $languageCode]));
106⫸ $action->setActionContext(
107⫸ new ActionContext(
108⫸ new ActionConfigurationOptions(['default_locale_fallback' => 'en']), // System context
109⫸ new ActionConfigurationOptions(['max_lenght' => 100]), // Action Type options
110⫸ new ActionConfigurationOptions( // Action Handler options
111⫸ [
112⫸ 'prompt' => 'Generate the alt text for this image in less than 100 characters.',
113⫸ 'temperature' => 0.7,
114⫸ 'max_tokens' => 4096,
115⫸ 'model' => 'gpt-4o-mini',
116⫸ ]
117⫸ )
118⫸ )
119⫸ );
120⫸
121⫸ $output = $this->actionService->execute($action)->getOutput();
122⫸
123⫸ assert($output instanceof Text);
124⫸
125⫸ return $output->getText();
126⫸ }
127⫶
128⫶ private function convertImageToBase64(string $uri): string
129⫶ {
130⫶ $id = $this->binaryDataHandler->getIdFromUri($uri);
131⫶ $file = $this->binaryDataHandler->getContents($id);
132⫶
133⫶ return 'data:image/jpeg;base64,' . base64_encode($file);
134⫶ }
135⫶
136⫶ private function getModifiedImages(): ContentList
137⫶ {
138⫶ $filter = (new Filter())
139⫶ ->withCriterion(
140⫶ new DateMetadata(DateMetadata::MODIFIED, Operator::GTE, strtotime('-1 day'))
141⫶ )
142⫶ ->andWithCriterion(new ContentTypeIdentifier('image'));
143⫶
144⫶ return $this->contentService->find($filter);
145⫶ }
146⫶
147⫶ /** @phpstan-assert-if-true string $value->uri */
148⫶    private function shouldGenerateAltText(Value $value): bool
149⫶ {
150⫶ return $this->fieldTypeService->getFieldType('ezimage')->isEmptyValue($value) === false &&
148⫶    private function shouldGenerateAltText(Value $value): bool
149⫶ {
150⫶ return $this->fieldTypeService->getFieldType('ezimage')->isEmptyValue($value) === false &&
151⫶            $value->isAlternativeTextEmpty();
152⫶ }
153⫶
154⫶ private function setUser(string $userLogin): void
155⫶ {
156⫶ $this->permissionResolver->setCurrentUserReference($this->userService->loadUserByLogin($userLogin));
157⫶ }
158⫶}
151⫶            $value->isAlternativeTextEmpty() &&
152⫶ $value->uri !== null;
153⫶ }
154⫶
155⫶ private function setUser(string $userLogin): void
156⫶ {
157⫶ $this->permissionResolver->setCurrentUserReference($this->userService->loadUserByLogin($userLogin));
158⫶ }
159⫶}


Download colorized diff

@mnocon
Copy link
Contributor Author

mnocon commented Feb 13, 2025

@adriendupuis image correctness improved in 4c323b9 (PHPStan is passing locally with these changes)

@mnocon mnocon requested a review from adriendupuis February 13, 2025 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants